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EVALUATION FOR DEGISION-MAKIMGi A CASE STUDY 



As the field of evaluation has developed^ it has become increasingly 
clear to its practitioners that it Is not and cannot be ba^ad on the 
same theory and methodology as more traditional educational research, 
Evalumtors do not operate in a research settingi their purpose is not 
to contribute to a general body of truth* Rather, they are addressing 
the specific concerns of a specific program or organization^ in specific 
ways. As awareness of this has grown there has been extensive discussion 
of what methodologies evaluation should use and of what theoretical 
basis is productive* Several models have been developed within which 
^valuators can conceptualize their task— and -their entire role. One 
such model which has been extensively discussed Is that of .decision-- 
b^sed evaluation. This model has been expounded by Stufflabeam et al 
and expanded j discussed or otherwise developed by a nimber of others 
(Guba^ Alklns $tc,) The basis of this model is that evaluation should 
provide Inf'jrmatlon which will be used by the decision makers in the course 
of their decision making. The appeal of this model to evaluators working 
in the field is obvious. If evaluations can auccegsfully be tied to 
decision making, their impact and worth are enhanced. If the purpose 
of carrying out an evaluation Is to provide information to decieion- 
makerSj theni the primary requirement is that the information lead to 
better decisions than would be made without it. Designs do not necessarily 
have to be experimental in nature, with random assignments and true 
control groups. There is an automatic focus for the evaluation— an 
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Important aspect, since few clients of evaluation are themselves capable 
□£ describing the focus they want. Within^ this very general model- 
really more of a philosophy~a number of mora detailed concepts must 
be developed, however. While this provides a genaral description of 
the goal of the evaluation, it does not provide much infoTOation about 
how to approach the evaluation itself. There are some materials available 
which address this question (those developed by CSE, for instance) ^ but 
the field is still badly in need of exploring. Just having a general 
goal, or even some specific ways to set up an evaluation design will 
not ensure that an effective evaluation will be set up. It is as difficult 
a task to ensure that evaluation results are in fact available to 
decision-makers and used by them as it is to set up a research sttidy 
In a school setting—and it requires eomnunication skills and often 
political skills for which most evaluators have little Initial background 
or training . 

The difficulty of goittg from a general statement of a goal to 
specific effective evaluations has been faced In detail in the Austin 
Independent School District, AISD set up in 1973 an Office of Research 
and Evaluation (ORE) I the baaic requirement from the district's viewpoint 
wae that several federal program evaluations should be carried out, 
ORE was iffMediately faced with setting up goals for itself and procedures 
for meeting those goals. Starting with a decision that evaluation 
should indeed serve decision-makers ^ ORE for its first year deeigned 



" evaluations In a frfflnework based on Stuff lebeam's CIFP model. These 
evaluations wsre carried out with a strong orientation toward defining 
the objectives of a program, then assessing the adequacy of the program 
iu meeting chase objectives* This approach led to a number of problems ^ 
however. Perhaps the largest was the lack of well^^formulated objectives 
for most of the programs ORE was called upon to evaluate* This lack 
meant that ORE evaluators had either to try to get the program personnel 
to specify objectives primarily for the purpose of allowing ORE to 
measure them, or they had to Impose a set of objectives written by ORE 
on the program. The results in either case had a tendency to represent 
rather trivial objectives ^ not usually related in any manner to the 
actual decisions being made by district and program personnel, 

Wiile believing strongly that the Ideal progr^ is objectives-- 
based j, formulated with careful planning to meet highly specific goals , 
ORE gradually realized that that ideal eKisted almost not at all in 
fact. Having realized this^ ORE decided to eKperiment with a different 
approach to evaluation* The evaluation chosen was one of the Implemen- 
tation of a new quarter system calendar and a new high school curriculum 
in the district 's high schools. The goal of the new calendar was quit© 
simple— meet the state mandate which called for every school in the 
state to be on a quarter system calendar by the 1975--1976 school year. 
There was really not any point in evaluating that goal— its achievement 
was obvious and trivial to document. The goal of the new curricula 
was essentially to provide every student in the district with the most 



appropriata aducatlon possible—so broad a goal it can*t be assessed 
at all w±thout*.a great deal of amplif icatiDn* In exploring the objectives 
of the prDgram with administrators It was clear that they were not 
truly interested in information about either of these general goals , 
nor were they able to generate specific objectives which did not sound 
either artificial or trivial compared to the program as a whole. 

It was also clears however, that there were some very important 
decisions which would be made during the coming year regarding both the 
quarter system and the curriculum^ and that the administrators were 
as yet undecided about these issues and would welcome information 
which would be relevant to them* For these reasons, a. different approach 
to evaluation was tried* Instead of focusing on objectives ^ the evaiuatioE 
was based on a number of decisions which were likely to be addressed by 
administrators during the year. These were phrased as "decision questions'% 
such as "Should the number of courses offered be reduced?" With a small 
amount of assistance, administrators found it easy to list many such 
decisions that they would be making. These decision questions were 
then related to "evaluation questions" reflecting useful information, 
such as "How many courses are actually offered?" The decision questions 
and evaluation questions together fomed a basis for an evaluation design 
quite different from the objectives=based design previously used by ORE* 
A summary of the design of this evaluation is presented below. 

The possibility of evaluating the quarter system first came to 
the attention of the Office of Research and Evaluation during 1973-1974 
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as a result of feedback from various district administrators regarding 
programs they felt most needed avaluation. The quarter system was high 
on tha list of programs which ware Indicated as raquiring assessment* 
Thus it seemed likely that the Office of Research and Evaluation would 
indeed be asked to conduct an evaluation of the quarter system. 

Having decided that an evaluation of the quarter system would have 
high priority for the following years the Office of Research and Evaluation 
began activities to design such an evaluation. During the spring, 
relatively little time was available to spend on this task; however, 
several steps ware carried out. A review of other district evaluations 
was undertaken In order to gain insight into the problems likely to 
occur in implementing our quarter aystem. In addition, meetings were 
held with the senior high principals and secondary coordinators to elicit 
possible decisions which would need to be made regarding the quarter 
systeni* These decisions were summarised and two lists of preliminary 
decision questions were prepared (see Figure 1 and Figure 2) * Finally, 
an estimatee! budget for carrying out a major evaluation of the quarter 
system was prepared and submitted to the district as part of the complete 
budget for the Office of Research and Evaluation* No further time was 
avaiiable before July 1 to pursue the quarter system evaluation design* 

The creation of the design continued during the sumner* The first 
step taken during this period was to review the dfaclsion questions 
formulated during the spring and prepare a list of possible data sources 
which might be of assistance in m&klng th*^ decisions (see Figure 3) * ■ 

■ . T : ■ 
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1. SiioiJ^d etaff deTelopment pro^ains be modified? If ao, iov? 

2, Should departmant ah&lraea ta alleeated mere plaanlBg tiaa? 
3» SliQiild departments orgasiEatian be mcdified? If ao^ haw'? 

U. Siciild metlisdi of eonmUEioatioa with parents he ahaaged? If so how? 

5^ Shoyld^ the eourse fair he repeated? If so^ ihould the isme foraat he used? 

6. Should the eo-ed P* S* program he modified? What ahaugea in reaponsihilities 
of teaehera need to he aade to accomodate aQ--ed claases? 

7. Should faeilltiea and aerviees provided to the achoola he modified to facili-* 
tate iTOlementatioii of the q^uaj'ter ayatei? 

8s Are additional per aonnel in aome adminiatrative poeitions reouired 'by the 
quarter eyate^? * , 

9. Should additional finanaial aupport he provided? 

10* Should pattern of aHoving e^anded eleotivea he ahanged? Should modif ieatione 
he aade in patterns of aounaeling atudenta in ohooaing Cowaea? 

11* Should the ina^eaaed level of aompeteneies he aoatinued? 

12* Shotid some couraei he eltoinatad or put on a fee haeis heoauae they pro- 
vide nothing ef educational or aoaial value? * . 

13* Should the raquir^eats for prerequiaite co^aea in areaa auah as health 
he modified? 

li** Sho^d the curriaulm he modified in light of hoard deaiaiona and mandates? 

15* Should the minlmuii alaaa iisa of 20 he loaaened aomevhat for one or two yeara 
to faeilltate quarter ayatem ir^laaentation? . 

l6. Should differtnt 'staffing patterns he i^lemented to deal vith different 
atudeat couraa-ahoiae patterns? 

17* Should the dlatriot provide tiranspartat ion from aahool to eehool and from 
sahoQls to" outside facilitlea to haek up the e^^anded aourse aelectlon? 



Figure li POSSIBLE DECISiON QUESTIONS-^PRINCIP^S. Decision quastlona for 
the quarter syacem evaluation generated by senior high school 
prlncipala in a meeting addreeeing poeaihle deaisiona to be made* 
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1> Siotild the giiErter lyitam return to omXy the eui^ieulira contained ia the 
smester lystem? Co we v^t the same sutstaative eam^ses in a reea^isable 
f ara? . 

2, ShoiLLd the ______ depai^eat alter its eoi;a*se offeringfl? If sa, ia 

vhat direction? Mditioai? Delttioos? Cottbiaatlans? Hevisions? 

3. ^ould the dCTartmeat ait^r its initructi©aal aethodologies? 
If BQ^ how? " " 

Shoiald subjaat area aad/or homeroom teachers fill the adviaa^ role? If so, 
idll additioaal training and materials %e needad? If not 3 what other altar- 
aatiTes might he tried? 

5. Should oourie faire ha ooatlnaad? If 10 ^ shoiad thay ha a^aaded, altered ^ 

--;-::;eto?--- * . 

6. Should procadin*as he eat up for additloflal QOordiaatioa hatveea Juaior aad 
ieaior highs (aSj for aKSS^ple, for q^uarter system choice preparatioa)? 

7* Cotirsa outliaasf 

a) Should they ha reTisad? 
h) If iQ, hy whom? 

c) If SO5 hy what process? 

d) If to 5 ia what diractloa? 

8, ShoiLLd thera he ravlsiona in the plaaement tasta? What kladg of ■ rayisioas? 

9* Should graduatioa requireaaatg ramla the mmm^ Should t hay ha chaagad to 
raflact ohangei in ausher of eleativai taken, ato? How? 

10, Should tha staadardiged teatlag program he altered? If so, how? 

11. Should the $lU05000 spant on materislis ato ^ he a ooatinuiag eMensa? Wiat 
should he tha leral of flaanelal support fort course guides rinsarricei 
priatiag; data pracessiag; etc^ Trm what sourae should tha support QOma? 
Should tha spaadiag patterns he iltaredf 

12* Should the procedm-e.^ for staffing Eftd asslpilag aowsas ha altared? ShovQ-d 
the staffing plan as em*raatly oawiad out ha coatiaued? Altered? 

13* What ahangas la school orgaai national pattarns appaar to ha naaassitatad hy 
the impact of tha tartar system. 

ik^ Should additional planning/ atteatlon oecw to facilitate high school/college 
iatarfaaa? 



Figure 2i POSSIBLE DECISION qUESTlONS— COORDIMTORS, Decision questions 
for the quarter system evaluation generated by secondary coord- 
inators (dla trio t^wlde supervisors of currlcultOT for specific 
subject areas). 
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1* Achievmeat 



A* CAT 

SAI aad CEB 
C * Gradei 

D. DAI? 

E. Departmental final i? 



2. Parent attitude 

A* Ptoae interrieva 

Written questionnalree vlth ptone "bactoip, 
C* Written quest lennalres irttli written tjaelmpp 
D, Student interrlevi ot -eaT'ents 

3* Student attitude 

A. Quest iennairts 
B* Interrievi 

Admliiian to, aeMevtment In^ eellage* • 

5^ Jo'b parforaance asd Jo'b acquiiltioii after finishing ichoQl, 

6* ^rollaent patterns. 

7* Teagher infomation 

8, Coungeler and other sahool perocsel iuformatian, 

9t A'bsenteelgm 

10. Stanmer ichool anreU^ents 

Hi Staffing requests and aeiignmants 

12. Information from U. T. admlaaiane persani 

13. Schedule of staff deTelapmeat offtrtDgi. 
lU, Pattern.a of aoOTiei 



a) getting enough students 

b) getting ataast enough students 
q) arerflairing 

d) ^ getting Tery f eif studentg 



Figure 3 1 POSSIBLE INFORMATION SOimCES FOR THE QUARTIR SYSTQf IVikLUATIONv 
Infarmation aourees suggested by admlniatrators or ORE far 
possible Inelusion In the evaluation design.. 
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This involved investigatirig what data were already available in the 

school system and considering what other kinds of useful data could 

be collected p Wien the tentative list of decision questions and possible 

sources of data had been prepared, a series of interviews was held 

with available secondary persomiel. This included the Director and 

Assistant Directors of Secondary Education and several of the secondary 

coordinators. Since it was sumer principals were not included at that 

time* The purpose of these meetings was to expand and clarify the decision 

questions 5 and discuss which data sources might be most useful to the 

secondary staff in reaching conclusions regarding each question* A 

s^ple of the form used for this review is shoTO in Figure 4* 
^^^^ 

Through the cooperation of two secondary coordinators^ a question- 
nalre was also administered to a group of teachers both to establish 
their priorities for the evaluation questions and to ascertain which 
data they thought most relevant to each question, ExMiples of the 
results from this questionnaire are presented In Figures 5 and 6. 

At the end of these reviews a preliminary design ^ consisting 
of tentative evaluation questions and methods for collecting data on 
these questions J was prepared * At this time, the quarter system ^.val- 
uation had not yet been funded , so work on the design was proceeding 
under Title III funds. Since the final scopr/ of the evaluation was 
unkno^m, the design remained highly tentative* The preliminary design 
was reviewed with the AISD Board of Trustees and with several secondary 
administrators. At this stage the quarter system evaluation budget 



qta^t«lf Sjae^f yoikdhtsE fair iteijlaa quasilesi, fl saelgg 



Figure 4: DECISION QUISTIONS WOHKSHIET* Form used for discussing possible 

decision questions for an evaluation with seGondary adminiitrators 
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contained in aemeatcj Syst^ 

^©Ul4 eeus'Se of r*rtn"gs - —-^ - 
be altered? ^ ^ 

^^tvee.. JP. ^ M^a? n=c 
^at te^^eas needed ia - L 
^^^^ 

WAe.^eat teats? 

^m^d OTduatl©3 reT4^e» ™ 
mestJ r^sia the sa^ef 

iboiild the it&sda^lied 
testis^ pi-eg^aa be alteredf 

What financial itjppc^ is 
needed f ^i^±Qtt3 ^eaaf 

ihe^y^d the prseed^e for ^ 
^at^alss csuratf be altered? 

/Wh&^ changes is. mchomX orgenl — 
latios pa'tter^ seeded? 

Feed addltioaal jls^iEg f , 
eollefe bo^d stydeatsf 

What ssdifiestlw needed' „ , 
u ecnE-se descriptisaaf ™; * 

If courses ^^t he sut, vhich '-z^ j 
should he eH^na^si? 
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Figure 5 TEAClffiR PRIORITIES FOR THE QUARTER SYSTEM EVALUATION , ^'Number of 
science and soelal studies teaehers indicating that each arma 
needed evaluation* • 
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ITEM I: Sh©ijld Che quareer ayaten rtcurn eiily €he curflculua eenlalneii lo, ths st^setr syacea? 
Ds «e want £ht isme subaCaBgive csursta In a ^egsgnliabla fsfml 



iOURClS_QF THFORMATON ; 

Teachar opinion of suecsaa oC ^sutsei in 
iseacin^ basic itudant 5asda« ^ • 
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30 



S£udeB€ adslsalona Co collage, 
StudesC opinion* t*.. 
Scudane advanced plkctmaiiSi 

itudanc su£€€as Im finding joba*...*. 
S Sudani attiCude Covard asho^l 
OtudeEfi self concepC.,^ *«•••■< 
P^iscipal opinion* 

Counaelor opinion^ ••^••^••c4«s«-*-**p 
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^.CmR- PRIORITIIS TOR INIORMMION SOURCES FOR^ THE SPlClf IID ; 
DICISION QUESTION, Number of teachers who felt that each source 
would be relevant" 111 deciding t^ether AISD , should return to ^ 
the - previous curriculum, : ; ; - ' 7 s ; ; : : 




Figure 61 



; was>pprov.d, and the scope of the; evaluation was astabllahad. A : 

complete draft design was then produced, whleh included decision questions, 
.valuation questions, overview of data collection and analysis . and ^ 
list of instrmnents With colUctlon times. This draft was reviewed / 
. by the secondary central office administrators, the senior high school 
principals, the secondary coordinators, the Evaluation Advisory Com- 
mittee, and various other persons. At the sa«e time, some Investigations 
into the cost and worth of collecting various kinds of data were begun. 

Finally, based on the input from various groups and the results 
of preliminary cost investigations, a final design was produced . This 
design reflected the same basic structure as the original design Mt 
had. however, been modified in a number of ways. For instance, a review 
of past studies of students who had graduated suggested that a great 
deii of effort was involved in eollecting relatively little Information. 
Thus. It was decided to conduct only a pilot study of former graduated 
this year, while concentrating more effort on preparing current students 
to respond to questionnaires after graduation. On the other hand, 
the perceived importance of teacher input was so great that a question- 
naire to be given to all teachers during staff development time was 
added to the sample^ teacher interviews originally planned. 

The final design was again reviewed by the Director and Assistant 
Directors of Secondary Education and by the senior high principals. It 
was then reviewed by the Cabinet and prepared for final distribution. 
The entire process of designing the evaluation occupied varying 



15 



13 



amounte of time from March to the end of September. A task analysis 
of time spent in designing the evaluation is presented in Figure 7 ^ 
The total cost to Office of Research and Evaluation of preparing the 
design was approximately $4500. This amount was primarily for senior 
evaluator time, secretarial time, reproduction costs, and other staff 
time. This does not include the cost of' the retime put in on review by 
secondary staff. Also excluded was the cost of instrument design and 
review, although drafts or final copies of most instruments were publish 
as an appendix to the design* An approKimate breakdoTO of costs is 
ehoTO below. ! 



1. Coordinator of Eesearch and &raluation - $ 300 

2* ienior Evaluator $3000 

3* Secrete^ , ' $ 500 

K-ocesi evaluators and data ipeelallsts $ 350 

5. Reproduction $ 250 

6* Other costs ^ inaludiag supplies 5 corauter $ 100 

■ time, etc. ^ " - 

Totii 4U50O 
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Figure 7 ; TASK ANALYSIS FOR QUARTER SYSTEM EVALUATION DESIGN. Est ma t es 

- of times required to complete various components of quarter 
systM design. * ; 



At the end of the first year of the quarter system evaluation 0S>^ 

was ready to review its eKperience and try again. * Several problems 

were liamediately evident* First p in the process of interviewing ad- 

miniatrators to elicit decision questions and InfortQatlon to be gathered ^ 

we had everemphasized some areas and underemphaslzed others* Administrators 

had difficulty epeclfylng Information sources or evaluation questions 

which would relate to their decision. Their Immediate reaction to 

"What would help you make this decision?" was principal opinioas teacher 

opinion^ achievement scores ^ usually in that order. On the other hand, 

too little Information had been gathered about the ttaes decisions would 

be made and the ttoe information would need to be available to impact 

those decisions. Secondj and more iiaportants was the aspect of which 

decisions to address * By the end of the year 0^ hadi 

* G'lthered data for decisions which were made before the Information 
could be provided/ 

CSathered information for which the decision was a foregone conclusion* 
, Gathered expensive information which provided only a minor part 

of the total used in making a decision, 
. Addressed some relatively minor decision questions effectively 

at the cost of omitting some more important ones* 

Finally j there was a weakness in the design Itself In that the relationship 
betpeen decision questlons™re£lectlng decisions to be made™ and eval- 
uation questions— reflecting Information to be gathered and analysed™ 
was not sufficiently eKpllcit * For some decision questions the evaluation 
questions were tangential and not particularly helpful. For others 
the most Important questions were overlooked* Thfe relationship between 
the two was not included in the design itself; It existed largely in 
the head of the evaluator* Thus It was difficult to ensure that the 



appro.^rlate Information was always provided* Furthermore, It would 
have been very difficult for a mm person to continue the ev^aluation. 

In address Ing these problems, ORE developed some much more specific 
guidelines and foraate used for the design of an evaluation. The basic 
purpoae of ore's evaluations is still to provide informatio<n for decision 
makers* The decision questions are designed to focus the evaluations on 
this aspect of ORE-s role, as well as to focus the audiences on the fact 
that evaluation is only worthwhile if some use is made of the results. 
They should represent actual decisions to be made by the personnel to whom 
they are addressed. 

The process of setting up decision questions for an evaluation normally 
goes through several steps. A tentative list of questions is originally 
created from a number of sources* These sources may include program staff 
and principals involvsd in the progr^s, administrative personnel who will 
be making decisions about the program, federal program officers, the program 
proposal (if any) , the evaluation proposals submitted to, the school board, 
and even the evaluator*s own perceptions of what questions are likely to come 
up related to the project. It is important to get input from the prtoary 
users of the evaluation data at this point, both to ensure that no important 
decisions are overlooked and to keep from specifying a lot of questions for 
which decisions have in fact already been made, or which will not come up. 
It is sometimes more effective to elicit decision questions in a group™ 
to some eKtent the comunication within the group will suggest some de— 
cisions no individual will think of, „ : 
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One problem likely to come up in the procegs of eliciting decision 

questione from personnel is that they will Include many questions which 

are really not decisional such as "How well are my stud^mts doing in math? 

These ar^ questioni to which they want to answer^ but they don't have any 

clear picture of an action tQ be taken as the result of knowing the v 

answer* The eVialuator is likely to have to continue asking, "^at do 

you do with that when you know it?" and "What kinds of actions are you 

going to have to take regarding the project?" before he gets real 

decisions. This process itself tends to be of value to the administrators 

of a programs since it forces them to give thought to the coming year 

and the issues which will need to be resolved* 

\toother problem ORE faces is that of "decisions" for which the person 

suggesting the decision question had already made up his mind, and no 

information the evaluator provides is going to change -it. In this case 

there usually Is a decision to be made j but the administrator doesn't 

want Inforaation to help him make lt"he wants a club to beat others 

into agreeing with him » Addressing such a decision may still be worthwhil< 

under certain circumstances; if the person proposing it is not the final 

declsion=maker and there is real controversy , it can still be useful. 

Otherwise it will exhaust evaluation resources without influencing 

anything one way or another* The evaluator must take care in the process 

of eliciting decisions to make it clear that he is gathering general 

ideas s not comit ting himself to anything anyone suggests, and also to 

- - .... • . . 

make it clear that he will not necessarily show that teachers really 

want this kind of materials or whatever. 

■ ■ t ■:..."-..■■„. 



Once initial decision questions are gathered > the nest step is to get 
some idea of what svaluation queations might be relevant to each deeisloh 
question. At the same tuae some attempt can be made to determine both the 
importance of the decision and the Importance of the information the evaluator 
can provide, in making that decision* The whols area of formulating relevant 
evaluation questions must involve the evaluator more directly; he knows what 
sources of data are readily available and probably will be able to specify 
many kinds of information which would not occur to program personnel* He 
mustj therefore^ take the initiative in this. The prioritiaing of both decisions 
and inf onnation is an important part of this step----people tend to feel that 
almost any information would be "nice", but to make effective use of evaluation 
resources^ the information must be not only nice but essential* 

The .third step is to select a tentative list of decision questions to 
be addressed* This selection will noraally be based on a number of criteria. 
Some of these are: Jjuportance of the decision; relevance of the information 
which could be collected to the decisloni evaluator -s perception ^if the 
likelihood of the Information being used If it is provided 5 etc, Obviouslyj 
the set of decision questions to be addressed needs to be Hjalted to those 
which can be answered by the resources available to the evaluation staff. 
In general. It is better to stick to & few big questions which represent 
Important J and as yet undecided j issues^ than to tackle a large number of 
■ smaller decisions. One consideration which must certainly enter Into the 
choice to be made is the timlng^ — ^caa the desired Information be gathered 5 
analysed and disseminated before the decision is made? If not, then 



there's not much point in Including the decielon question. At this 

point, the ^evaluat or should have enough feel for the evaluation questions 

. ^ ... 

Involved to have some idea how well they provide information eoncernlng 
the decision question and how expensive it is going to be to answer them; 
these definitely also enter into the choice* r-^^ 

The office has tended to get the most response to its evaluation 
information when it really does represent something the decision maker 
doesn't already know, Thus^ principals generally believe that they 
have a fairly good feel for teacher attitudes in their schools | a. teacher 
attitude questionnaire is most likely either to confirm their feelings 
or to be rejected; student pattern of course choice (for example) , 
on the other hand, is less likely to be at their fingertips. This 
also enters into choosing decision questions, in that the latter type 
of information is more likely to result in a different deeislon than 
would be made without input from ORE, Note^ however, that It is sometimes 
legitimate to provide information simply to have ORE 'a ''stamp-* on it ^ 
even if it's something everybody in the district knows. One final 
point-ttie evaluator should allocate some re^sources to one decision question 
"to be determined at a later date". Things do come up. The worksheet ORE 
utfes lii" making these determinations is shown In Figure 8 # 

When the decision questions have been selected, the evaluator 
drafts an initial design. This design includes a sumary of decisions 
addressed, a specific chart of evaluation questions tied to each decision 
question, and an allocation of resources to each information source 
being included (Resource allocation is another major deelgn issue—but 



that's a different paper*) The design is formulated to deal with several 
pf the problems ORE encountered the first year. By making explicit the 
relationship between decision questlone and evaluation questions it is 
easier to ensure the appropriateness of the tie* The date the information 
must be provided is included in the list of evaluation questions. The 
resource allocations can be easily reviewed to decide whether they are in 
accord with the importance of the information being gathered, Reaources 
are allocated primarily by instr^ent or Information source^ORE has found 
that it is easier to structure day^to^day activities in terms of data to 
be gathered and analyaea to be performed^ rather than in terms of reports 
to be made* , 

At this point the evaluator has a tentative designs including a set of 
decision questions which are within fhe scope of his resources to provide 
InforTnation on. The last step is to review them with all the persons from 
whom they were originally elicited^ as well as some persons (school board?) 
not originally involved. The evaluator is prepared at this point to let 
people know why he left out their favorite decision questioni If someone 
wants to add a question he makes it clear that another has to come out* 
He is also prepared to deal with the situation of an administrator insisting 
there is no decision to be made (or that it has feeen laade) when it is clear 
from his superior that the issue is far from r^-s^lved* At the end of the 
reviewp the evaluation Is underlay for another year. An eKMple of a finished 
design is shown in Appendix A. 

In ore's experience with this approach we do not feel that ORE has 
found any magic key that toU make our evaluations great, our administrators 



whigzesi or even all our activities- worth doing. Indaad, there remain 
a variety of problems. One major problem hat been well illustrated by 
Alkln— all evaluation activitiis occur within a general context which include? 
expectations on the part of the decision-maker of what evaluation is about. 
If h^ doesn't perceive evaluation as providing information. for his decision 
making, there is little likelihood of It serving that role, no matter what 
the evaluator perceives. Another which we have been repeatedly made aware 
of and which Alkin also addressei, is that of providing information in time 
to be useful, A third, which most evaluators are by now well aware of, 

...... • • ■ . .... - • m _ 

is a large committment of resources to delineating and obtaining Information, 
which leaves few resources for providing it. This. latter is aggravated by 
a lack of understanding on the part of funding sourcea for the complexity 
and expense of this last task™ORE's data processing budget, a very technical 
area, is seldom. challenged, but reproduction expenses often seem to ™aze 
our funding sources , We have also found specific settings which seem to 
make a decision questions approach much more difficult to implement. If 
a program has mandated ob j ectives , the evaluation usually must address them . 
If it Is to address decision questions as well, this leads to a sort of 
hybrid design which can be awkward and time^-consuming to work with. If 
an evaluation is for needs assessment purposes. It can be difflcnlt to 
dden t if y relevant decision makers and very difficult to identify decisions 
to be made. This format of evaluation design is very sensitive to problems 
with the decision making process in a district* ORE has found, for instances 
that many decisions are Mde unexpectedly at meetings | if we are there ^ " 
our inforMtion is . considered; if not, it isn' t. Many decisions in AISD 



are also made well before the setting in which they are formalized—over 
coffee^ in other meetings^ in a hundred Gasual contacts in which the matter 
is diacussed. If ORE can provide information in these infomal settingSj 
it is very likely to be considered. If the information comes only to the 
formal meetings it may well be too late* 

Nonetheless 5 we feel that we are much more effective than we were 
four years ago in dealing with this area* Our information has had an impact 
on decisions. We hear from administrators and board memberi information 
we gave them as the reason for a decision* Occasionally someone even calls 
us before a decision is imminent and asks us if we have any relevant 
information"certainly a change from four years ago. There remain many 
problems^ but we feel that this model is a good one. We hope that by sharing 
some of our experiences we may contribute to those who face all the same 
problems and tribulations we have. We also hope to stimulate continued 
discussion of specific techniques for Implementing a decision^maklng model 
of evaluations with the hope that someone else can contribute to the solution 
of some of our problems J 
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APPENDIX A 



FmUATION T5ESinN 
1976^1977 

ninn .school cmniCirLm EVALUATTnN 

November 5, 1976 



Stan Drezek ■ 
denier IvaluatQt^ 

limine Morris Tom Roudebush Barbie Ballinghausan 

Evaluation Assistant Evaluation Assistant Secretaiy 




: : :: Publication _:_ : : ■ '. 



PROGRAM STAFF 



The followliag Austin Independent School Dla£r±ct .staff nrabers are respon- 
sible for the Impl^encatlon of the major aapeots of Che high achool , 
curriaulTO evaluated la thia design » JlQwmwmv^ thera are mmny persons la 
the Individual high schoola and central administration not llated who dlraetly 
affeet^ t!4e..aurrleulum— to partleular all. the high eehool teachers, eoimaalarag 
and depa^^tment heads. 



ar. Bill Smoot 
Dlreetor of Secondary Education 



La^enee Buford 
Assistant Director of S 
Education 

Principals 

Charles Aklns 
Jacquelyn McGee 
Forest Kline 
Ron Beauford 
Adan Salgado 
Rodger Wiley 
W*A, Sloan 
James Vlramontes 
Jack Allison 

Coordinators 



La Rue Mllsdn 
Ida Courtney 
BlimLineTO__ 
Curt Eckardt 

Ron Foy 
Sherllyn Howze 
Loyce Iga 
Myrtle Johnson 



Vera Jones 
Bab McLean 
Bertha Means 
Julia Mellenbruch 
Franeea Nesmlth 
John Pearson 
Hargarst Ruska 
Wayne S chads 
Elgin Schllhab 
Richard Sutch 
Perry Suter 
La Neal Tankarsley 
Don Ilrkla 

Imterna to Coordlnatora 

Sus Fleming 
Anna Floraa 
Irene Kanter 
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Maud Sims 

Assistant Director af Secandary 
Iducatlon 



Anderaon High Schaal 
Austin High School 
Crockett High School 
LBJ High School 
Johnston High Schaal 
Laniar High Schaal 
McCallum High Schaal - 
Reagan High Schaal - 
Travis High School 

AREA 

Hamemaklng Educatlan 
Library Services 
Jacatlan^^aparaclve^Tralntng—— — — 
Phys, Ed, I Health, Safety and 

Elver's Ed, 
Industrial Arts 
Art Iducatlon 

Trade and Industrial Vocation 
Business Ed, , and Vacatlanal Office 

Ed* ^ 
Student Activities 
Learning Resources 
Reading 

Foreign La .iguage 
Saclal Studies 
Instrumental Music 
Language Artr 
Science 
Bteth^atlcs 

Choral Music Education 
Pre*-Vaoatlonal Industrial Arts Ed , 
Girls Physical Education 
Ifedla Services 



Selence 

.Mathematics 
Social Studies 
LanMaee Arts 
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EVALUATION DESIGN REVIEW FORM 



I have reviewed the evaluation deslgA for the High School CurTleul™ Evil-' 
mmtlom, I eomcur that the deelslon questions are appropriate^ that the 
progran description as stated Is representative of our prograin, that 
the evaliiaclon questlofis are appropriate^ and that the types of Instrtffiiemts 
proposed are acceptable « 




Dlreetor of SeMndary Education 



^^Istwt Director of Secondary 
Education 




islstant Director of^econdary 
Education 



FriAclpalp Anderson High School 



F^ncli^lp Austin High School 



tr^^toal, Crockett High School 




PrlnclF^^ JolmstDii Hlgh^ School 

106I 



Prlg^ffialp Lanier Wgh Scho^ 





Ppteplpalp McCalltTO High School 



Ipalp Reagan High School 



Frtoclpal, Travis High School 
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DOCISION QUESTIONS ADDRiSSiD 



A. Systain^Level Decision Questions 

1, Given student perfomanee on a range of autcooe TCaaureSp in what^' 
If any, areas should the revieed currieulia undergo major change? 

2, Should the quarter course credits and fflin^rom reading and mathematics 
eorapetenoles required for graduation be changed? 

3, (Deleted) 

4, Should AISD modify the job descrlptlan and responsibilities of high 
school counselors? 

5, Should AISD move toward year-round schooling? 

B, Program-Level Decision Questions 

6, What changes need to be made in the advising process to assure that 
students select appropriate courses? 

7i How can the junior high schools be assisted to itaprove the orientation 
of alghth graders to the high school curriculum? 

8. Should AISD consider acting to Incraase the stoilarlty of the same 
course taught In different schools by dlfferant teachers? 
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DICISION QOiSTIONS OVERVIEW 
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DECISION QUESTIONS OVIRVIiW 
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PROGRAM SUMMARY 



The 62nd session of tha atate legislacure passed in 1971 a law mandating ' 

Che liBplementaElon of the quarter ayitam for all Texas s 

The law required districts to operate the equlvale^^ of three quarters 

and to provide 180 days of Instruction* tte rationale for the law 

was both econonilc and educational. Operating a fourth quarter could save - 

tax money by utilising the existing physlca meet Increased 

enrollment without new constructloni also. It eould give both business sad 

students greater amployment opportunities * Educatlona the quarter 

system could increase student involvment In' choosing their frlari of study 

and present a greater range of courses for salactloii. 



Work on Implementing Che quarter system in the Austin Indepe 
District began in 1972 under a Secondary Study Conroit tea. Our district 
desired to use the quarter system as the organizational vehicle for curric- 
ulum revision aimed at individualization, A Quarter System Steering 
Committee was subsequently appoint^^^ and it eatabllshed task forces of 
department heads ; and instructional coordinators to develop the new 
curriculum. As of February 1974 ^| 700 possible courses had been identified, 
-^intensive effort was begun to write course descriptions jdeterolne pre- 
requisites and grade-offered, set graduation requirements, and e^ 
overlap among the courses. The result was a new currlcu^lum allow^^^ 
students to develop plans of study tailored to their IndiSfidual needs and 
interests, yet meeting the graduation requlrementaestabllshe by the 
State Boart of Education and the Texas Education Agency, - 

Gurrently the Alls tin Independent School District of f era a high school ; 
program of three quarters of free public education. These are three 12-week 
60-day periods oceurlng within the traditional late^^^ to end*Kif-May 

semester school year. There is an optional fourth stmMr quarter students 
can attend for tuition. This qm ier offers the possibility of an 
accelerated graduation (the graduation requlrwfnta can be completed in 3 
calendar years) * remedial workj or enrichment. 

This quarter system program is delivered through nine conprehenslve high 
schools ranging In student membership from' about 1300 to 3000 and employing 
from about 70 Co 140 teachers. The median percent student attendance for 
these high schools In 1975-76 was 89 • the typical student was present 
about 53 days per 60-day quarter. The pupil-teacher ratio for these schools 
ranged from about 18,6 to about 22 :4 ; however ^ the class-slEe In core sub- 
ject areas was closer to 28-30, Only one high school~had a "large" per- 
centage of students from low-Income familles^(49Z) | the other eight high 
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schenls ranged from 4% to 20%. Four high schools were vary near 80% or 
more anglo in itudent membership r one hlp,h school was pradominan 
minority i while the rewalnlng four high schools had a greater etJmlc 
diversity. These 1975-76 school charaeterlstlrs will probably be close 
to the 1976-77 data* but laay change in 1977-78 with the possible onset 
of- buelng* .• 

The currJculum ttselF consl^tefl in 1975-75 of 698 qu.irter coursp^. One 
hiph school offered as many 558 of thrRf? cotfrses nml nnothrr n?^ 

few ns (64%) nf the courses over thr thref'-qLinrtef year / nbyiinii^ly 

size nf the hinh school flttermln^d some? nf the v/ir fnb I L t ty Jn course 
offerings. 



To administer this curriculum the district hflj^ ^^n ^flnilnlstr/itive stnff 
consisting of a director of Secondary Mucnttnn, two assistant dlrectnr.^* 
nine principals » and approximately 20 Missis tnnt principals In addir. inn 
to other support staff. To Implement the currft?fi1um 21 secondary 
InRtructlonal coordinators work wl th a staff of nrarlv 1000 teachers 
and department chairman. 

The actual program, the hlph school cv*^^* rum, i s the full ^et of cnnrRes 
offered. This program is presented the nnxt pn^e; in abhrevtated farm. 
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I. RetfMnct far this UitlflR is Ehg "1976-7? I!lf,Ii BchooLInfijrTintiQn fiuldi" 
bi Iiteludii luiingii Idw, Ruilnais Hanagei!iatit, Advertising, Offlea 
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EVALUATION SUMMARY 



The 1975-6 Ouarter System Evaluation foeusad In Its first year on how 
well implementation of the quarter system had proceedad and on gacherlng 
bafeline data for futur- examination of the effects o£ the quarter system and 
revised curriculum. The Board, after reviewing propoaea'evalultflona" 
for 1976-77, decided to continue high school avaluatlon* 

The focus for 1976-77 shifted to eaphaalM the curricula a ^ 
outcomes, more than quarter ays tarn impl(im#ntat^on. The focua has alao 
shifted to use of "hard data" whsrever this la available, with aubjeative 
opinion data sharply limited,; Given somewhat reduced funding It has been 
necessary to limit the areas considered to t ho ^© which appear most vital. 



Information regarding the hloh school curriculum will be collected pnd 
sented in chase general areas* 



. Student performance^ includinp longitudinal Information on achievement 
scores as measured by the Sequental Tests of Education Prqgres^ and 
Scholastic Aptituda Test (CEEB) , attitudes as neasured by tha Survey 
of Stiidy. Habits and: Attltudesj at tendanca, grad rates, / 

and judgements of relevant staff ' V- ■ ^ 



2. The advising process ^ including both continuing questions on who carries - 
out the advising, whether students are adequately advised and whether 
course selections are appropriate, and adding conalderation of t^e 
advising process in the junior high schools, 

3* Ofaduation requirements , including ppoj acted effect gf competency 
requirements, comparison of AISD seniors achievement scorep to a 
national sample, how students are using elective courses, and wh^t 
courses are actually taken. ' 

Organizational roles ^ Including study nf rhf> duties , responsibllt^las, 
and perceived benefits of counselors, as well as the perceived "coats" 
of expanded curriculum implementation* 

^* Course content ^ includlnp. how similar the same course taught by 

different teachers is, ■ - 
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INFORMATION SOURCES SUMMARY 
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SUMMARY OF DATA TO BE COLLECTED IN THE SCHOOLS 



A, STUDENT DATA 

Week of STUDENT ADVISING CREC^lSTt Adminiatered to a rmndoa 

Occ, 11 laaple of 38 sacclans repreienting all dlitrlet 

nlnth-gradars * CQmpleta admlatstration directions (script) 

will bs provided. 

Weeks of COUNSELEl nuKTlONHAIMr Admiaiatered to a rasdo© B^mplM 

Nov. 29 °' about 36 lOchj 11th and 12th grade social studies. 

PgQ^ $ sections by 0BJ2 staff .""Requires about 30 minutes. 



Weeks of 



STUDENT QireSTlOraAIREi Adminlatered to 30-40 classaa 
, (rando^y selected) rapresenting each grade level and 

each high school by ORE staff. Requires one-half period. 

Mar i d>k 

Mar- 28 - 

Weeks of ^ SURVFf OF SinJDY HABITS ATTITUDES: Administered by 

ORE staff to a randomly selected sample of about one 
^ ^ class at each grade level in €aeh high school. Requires 

21* ^ full class period* 

Mar. 28 

Throufthout EARNED CREDIT RimVtYr Collected by ORE staff from 

entire year available records In each high school . This may 

Involve one or two days with one or two nRE staff 
working at each high school; to he Individually arranged. 



Note I The SEnUENTIAL TEST OF RDUCATTONAL PROnRESS will be 
administered In mld-Aprll under OHE's dlptrict--wide 
cesclnn. 
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B* TEACHER DATA 



Week of THSTS AND K^IKAttonS r Copies of all foraal Cases 

Nov. 22 and examinations will b© colleccad from 2 raiidQmly 

seltccad sections of one salecced Lasguage Arts, 
Social Studies I and Science coursB at each high school 
Requires teachers to subnilt copies nnly, 

W^^^ TFACHER PHfTIO^mATREl Administered to all secondary 

J*^* 17 teachers by H SB' mall by ORE staff. Requires about 

25 minutes. 



C. NON-TEACHING FACULTY DATA 



Week of IJnN-TEACHINC HTCH SCHOOL PERSONNEL nUESTIONNAIH; i Ad- 

Jan, 17 ministered to all assistant principals, deans, counae- 

lors, reglstrarSj secretarlesp librarians , and attend- 
ance clerks by AISD mail by ORE staff. Requires about 
20 minutes^ 



Week of COUNSELnR INTERVTEW/nUESTinNNAIREi Interview of one 

Jan* 17 (Q,) randoTnly selected counselnr at each high school by. ORE 

Feb, 21 (I,) staff at the inten/lewee*s convenience. Requires abou^ 

one periodi the other counselors will be surveyed, 

quires about 30 mlnutei* 

Weeks of PRINCIPAL INTERVIOT^ Interview with each senior hl^jh 

Feb* 14, school principal by a Senior Evaluator, Requires 

Feb, 21 about one hour. 
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EVALUATION TIME RESOURCES ALLOCATION SUMMARY 
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